(19) 



J 



Europaisch s Pat ntamt 
European Patent Offic 
Offic uropeen des br v ts 



(11) 



EP 0 896 483 A2 



(12) 



EUROPEAN PATENT APPLICATION 



(43) 


Date of Dublieatiorv 


(51) Intel e H04N 17/00 




10.02.1999 Bulletin 1999/06 






(21) 


Application number; 98306092.2 








Rot a rA filirirr 1ft CY7 1 QQfi 
UaT6 Ol Tiling. OU.Ut. I a3u 






(84) 


Designated Contracting States: 


(72) 


Inventors: 




AT BE CH CY DE DK ES F! FR GB GR IE IT LI LU 


• 


Isnardi, Michael Anthony 




MC NL PT SE 




Plainsboro, NJ 08536 (US) 




Designated Extension States: 


• 


Dieterich, Charles Benjamin 




AL LT LV MK RO SI 




Kingston, NJ 08538 (US) 






• 


Baumgartner, Hans Andreas 


(30) 


Priority: 30.07.1997 US 902965 




Hamilton, NJ 08690 (US) 


(71) 


Applicant: Sarnoff Corporation 


(74) 


Representative: Pratt, Richard Wilson et al 




Princeton, NJ 08543-6449 (US) 




D. Young & Co, 








21 New Fetter Lane 








London EC4A 1DA (GB) 



(54) Testing a video decoder 



(57) An encoded bitstream for a generator (110) is 
applied to a black-box video decoder (120) to evaluate 
the performance of the decoder (120). The encoded bit- 
stream is designed such that the output from the decod- 
er (120) can be evaluated without the need of a human 
observer. The encoded bitstream is applied to both a 



decoder under test (DUT) (120) and a compliant refer- 
ence decoder (130) and a video output analyzer (140) 
compares the results from both decoders (120, 130) to 
determine compliance of the DUT (120). Additional em- 
bodiments address postprocessing and the selection of 
a subset of test frames for evaluation. 
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D scription 

[0001 ] The present invention relates to a method and 
apparatus tor evaluating a decoder. The present inven- 
tion also relates to a bitstream for evaluating a decoder. 
An illustrative embodiment of the invention relates to a 
method and apparatus for evaluating the decoding of 
video signals by a "black box" video decoder. Preferred 
embodiments of this invention relate to an automated 
method and apparatus that evaluates the functionality 
of a "black box" video decoder without the need of a hu- 
man observer to judge the quality ot the decoded video 
signals. 

[0002] The increasing development of digital video/ 
audio technology presents an ever increasing problem 
of resolving the inter-operability of equipment of differ- 
ent manufacturers. Although various organizations, 
such as the Moving Picture Experts Group (MPEG), cre- 
ated various I SO/I EC international Standards, e.g. 
1 1172 and 1 381 8 (generally referred to as MPEG-1 and 
MPEG -2), to establish a standard coding/decoding 
strategy, these MPEG standards only specify a general 
coding methodology and syntax for generating an 
MPEG compliant bitstream. Thus, many variations are 
permitted to accommodate different applications and 
services. This flexibility creates product differentiation 
in the marketplace, where it is difficult to ascertain the 
compliance and quality of an alleged compliant device, 
e.g., MPEG-like decoders. 

[0003] Unlike analog television systems, digital de- 
coders are highly non-linear and often contain memory. 
A digital decoder may operate normally over a certain 
range of a certain parameter, but may fail dramatically 
for certain other values of that parameter. Furthermore, 
a digital decoder may depend on previously decoded 
data for proper behaviour. 

[0004] Typically, the testing of complex digital sys- 
tems such as decoders is performed by stimulating the 
decoder under test (DUT) with a known sequence ol da- 
ta, and then analyzing the output data sequences or the 
intermediate data sequences using, e.g., a logic analyz- 
er, to determine if the results conform to expectations. 
Although this is an effective testing technique, it requires 
xtensive knowledge of the circuit implementation or ob- 
servation of internal nodes of the particular decoder. In 
fact, a human observer is typically needed to evaluate 
the output from the decoder under test to determine its 
compliance. 

[0005] However, in many instances the decoder is a 
"black-box" that accepts a bitstream (encoded signal) 
as input and provides a digital or analog representation 
of the decoded signal as an output. Due to product dif- 
ferentiation in the marketplace, it may not be possible 
to acquire such technical information for all decoders. 
In fact, ev n if such technical information is availabl , it 
may not be cost eff ctiv to construct a different test se- 
qu nee for every decoder. 

[0006] Therefore, a need exists in the art for a method 



2 

and apparatus for testing decoders (for example MPEG- 
like decoders) without prior knowledge of the particular 
circuit implementation of any particular decoder. More 
specifically, a need exists for an automated method and 
5 apparatus that evaluates the functionality of for example 
a "black box" video decoder without the need of a human 
observer to judge the quality of the decoded video sig- 
nals, e.g., whether a DUT is in compliance with a par- 
ticular standard. 

[0007] Aspects of the invention are specified in the 
claims to which attention is invited. Features of depend- 
ent claims may be combined with those of other depend- 
ent claims and/or with independent claims as appropri- 
ate and in combinations other than those explicitly set 
out in the claims. 

[0008] An embodiment of the present invention pro- 
vides a method and apparatus for creating and applying 
an encoded bitstream that represents one or more test 
frames, to evaluate the performance of a black box vid- 
eo decoder. The encoded bitstreams are preferably de- 
signed such that the results from the DUT can be eval- 
uated withoutthe need of a human observer. An embod- 
iment of the present method applies the encoded bit- 
stream to both a DUT and a compliant reference decod- 
er and compares the results from both decoders to de- 
termine compliance of the DUT The embodiment of the 
automated method preferably evaluates the differences 
of the decoded luminance and chrominance signals be- 
tween the DUT and the reference decoder on a frame 
by frame basis, which may include one or more levels 
of analysis, e.g. pixel by pixel, block by block, macrob- 
lock by macroblock or etc. 

[0009] Furthermore, an embodiment of the present 
encoded bitstream is designed to effect easy frame and/ 
or pixel synchronization of the DUT and the reference 
decoder. Namely, the synchronization ensures that a 
comparison is performed on the same frame at the same 
horizontal and vertical position. This synchronization is 
achieved through the use of "key blocks", which are 
blocks having a specific image pattern. 
[0010] In one embodiment, the encoded bitstream is 
designed such that only a subset of the frames within 
the encoded bitstream is actually compared, thereby in- 
creasing efficiency and speed of the test and reducing 
the storage requirement necessary to store a large 
quantity of video frames. 

[0011] The present invention may be more readily un- 
derstood by considering the following illustrative de- 
scription in conjunction with the accompanying draw- 
ings, in which: 

Figure 1 depicts a block diagram of an illustrativ 
test apparatus according to the present invention; 
Figure 2 depicts a detailed block diagram of the vid- 
eo output analyzer and its relationship with th DUT 
and the refer nee video decoder; 
Figure 3 depicts a test frame having an intra coded 
area and an actual t st data area; 
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Figur 4 depicts a t st frame having k y blocks; 
Figur 5 depicts a numerical format for displaying 
errors detected by the video output analyzer; 
Figure 6 depicts a colour format for displaying errors 
detected by the video output analyzer: and 
Figure 7 depicts a flowchart of an illustrative com- 
pliance testing method according to the present in- 
vention. 

[0012] To facilitate understanding identical reference 
numerals have been used where possible, to designate 
identical elements that are common to the figures. 
[001 3] Figure 1 illustrates a block diagram of an illus- 
trative test apparatus 100 according to the present in- 
vention. The test apparatus comprises a test bitstream 
generator 11 0 : a video decoder under test 120, a refer- 
ence video decoder 130, a video output analyzer 140 
and a display 150. 

[0014] In the preferred embodiment, the test bit- 
stream generator 110 comprises a general purpose 
computer (or application specific integiated circuits 
(ASIC)) having a central processing unit (CPU) 114, a 
memory 11 2 and a plurality of Input/Output (I/O) devices 
116. The test bitstream generator 110 generates a plu- 
rality of encoded video bust reams that arc applied to 
both the video decoder under test 1 20 and the reference 
video decoder 1 30. The encoded video bitstream can 
be stored in the memory 112 or other storage devices 
(I/O devices). The I/O devices may include, but are not 
limited to, a keyboard, a mouse, a camera, a camcorder, 
a video monitor or storage devces. eg. a hard disk 
drive, a floppy drive or a compact disk drive. The input 
devices serve to provide inputs to the test bitstream gen- 
erator for producing the encoded video bitstreams or to 
allow the test bitstream generator to retrieve the encod- 
ed video bitstreams from a storage device 
[001 5] The video output analyzer 1 40 receives the de- 
coded video output signals from both the video decoder 
under test 1 20 and the reference video decoder 1 30. By 
comparing and analyzing these two output video sig- 
nals, the video output analyzer 140 is able to determine 
whether the DUT is in compliance with a specific stand- 
ard, e.g.. the MPEG or Advanced Television Systems 
Committee (ATSC) standards. The preferred encoded 
bitstreams of the present invention arc designed to pro- 
duce video output signals that a re used by the video out- 
put analyzer to determine compliance ol the DUT, there- 
by eliminating the necessity ol a human evaluator. 
Namely the preferred embodiments ol the present in- 
vention does not require a human to look at the decoded 
test frames, and to judge (or to detect specially designed 
patterns) whether the displayed video frames from the 
DUT are compliant to a standard In turn, the results 
from the video output analyzer 140 can bo displayed on 
a monitor or display 1 50. 

[001 6] The DUT 1 20 can be a physical video decoder 
devic which is coupl d to the test bitstream gen rator 
110 through a communication system such as a trans- 



mission channel. Alt rnativ ly, the DUT can b r pre- 
s nted by a software implementation (application) re- 
siding in the memory 1 1 2 of the test bitstream generator. 
[0017] Similarly, the reference video decoder 1 30 and 

5 the video output analyzer 140 can be physical devices, 
e.g., using general purpose processors with memory 
142 or custom ASICs. However, both the reference vid- 
eo decoder 1 30 and the video output analyzer 140 can 
also bd represented by software applications residing in 

io the memory 11 2 of the test bitstream generator. 

[0018] FIG. 2 illustrates a more detailed block dia- 
gram of the video output analyzer 140 and its relation- 
ship with the output signals from the two decoders. More 
importantly, FIG. 2 also illustrates an alternative embod- 

15 iment where it may not be possible to directly access 
the decoded test frames from the DUT In some situa- 
tions, the output signals from the DUT may represent 
video signals that have undergone one or more post- 
processing steps, e.g., frame rate variation due to 3:2 

20 pulldown, pan-and-scan and other display functions. 
Since these display functions are not restricted by th 
MPEG standards, they are not required for compliance. 
These additional layers of postprocessing steps in- 
crease the difficulty of ascertaining the compliance of a 

2S particular DUT A novel method to address this criticality 
is provided below. 

[0019] Furthermore, although an embodiment of th 
present invention is described below with reference to 
an MPEG standard, it should be understood that th 
30 present invention can be applied to other MPEG-based 
standards, including but not limited to. ATSC. DVD and 
DVB or similar standards such as MPEG-4, H.261 and 
H.263. 

[0020] Returning to FIG. 2, a basic set up for testing 

35 compliance of MPEG-like decoders is illustrated. A spe- 
cially constructed encoded input test bitstream enters 
the DUT 120 via path 205, and, if possible, all decoded 
output frames from the DUT are captured via path 207. 
The DUT produces one luminance signal (Y'[i,j;k]) and 

40 two chrominance signals (Cr^ijik] and Cb'Ii.jik]), where 
i and j are the horizontal and vertical pixel indices within 
a frame and k is the frame index. 
[0021] The input test bitstream is also received and 
decoded by the reference video decoder 1 30 which is 

4S known to be MPEG compliant, such as the latest version 
of the software video decoder, "mpeg2decode". This 
video decoder was created by the MPEG Committee 
and is generally available on the Internet at "www.mpeg. 
org" Since the encoded input test bitstreams and the 

so reference video decoder 1 30 are generally available be- 
■ forehand, the decoded frames (Y[i,j;k]), Crti,j;k] and Cb 
[i, j; k]) from the reference video decoder 1 30 via path 209 
are optionally decoded in advance and are stored in a 
storage device (reference source), such as the memory 

5S 112 or 142. Storing th output vid o signals from th 
referenc video decod r 1 30 prior to p rforming a com- 
plianc test on a DUT. incr as s efficiency and reduces 
computational overhead, which is especially important 
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in "real time" application. 
[0022] Th decoded frames from both decoders 1 20 
and 1 30 are received by the video output analyzer 1 40, 
where on a frame-by-frame basis, the Y, Cr and Cb ma- 
trices are differenced (e.g., pixel-by-pixel), and the mag- 
nitude of the error is computed via a Y compare section 
220, Cr compare section 222 and Cb compare section 
224 respectively. 

[0023] It should be noted that slight round-off differ- 
ences are found in the implementations of the Inverse 
Discrete Cosine Transform (IDCT) in different decoders, 
such that the pixel values may differ slightly, yet the DUT 
may still be compliant. According to clause 3.3 in the 
"IEEE Standard Specifications for the Implementation 
of 8x8 Inverse Discrete Cosine Transform" (IEEE std 
1 180-1 990), for any pixel location, "the peak error (ppe) 
shall not exceed 1 in magnitude." The MPEG -2 standard 
cites this document in Annex A. The value "1 " is in units 
of brightness defined by the MPEG standards, e.g., 
where the color "white" has the value 240 and the color 
"black" has the value 16. Thus, the Y, Cr and Cb com- 
pare sections perform this absolute error function and 
permit the video output analyzer 1 40 to determine com- 
pliance in an objective manner without the need of a hu- 
man evaluator. 

[0024] The selection of a particular peak error thresh- 
old is often associated with a particular standard in 
which a DUT is being evaluated. However, a different 
peak error threshold can be selected for other reasons, 
e.g., to provide a slight tolerance in testing. Thus, it 
should be understood that the peak error threshold can 
be adapted for different applications. 
[0025] Returning to FIG. 2, a postprocessing section 
(postprocessor) 210 is typically incorporated in a com- 
mercially available MPEG decoder system. The post- 
processor may, for example, upsample chroma to have 
the same horizontal and vertical size as luma and con- 
vert from Y, Cr, Cb to the Red, Green, Blue (R, G, B) 
color coordinate system for display purposes. The post- 
processor may also perform 3:2 pulldown, pan-and- 
scan and other functions. As discussed above, since 
these display functions are not restricted by the MPEG 
specification, they are not required for compliance. 
[0026] In a second embodiment of the present inven- 
tion, compliance of a DUT is determined by evaluating 
the output of the postprocessing section 210 (or other- 
wise known as display processing section). This second 
embodiment is necessitated by the fact that it may not 
be possible to capture the output signals from the DUT 
on path 207, as illustrated in FIG. 2. For some DUTs, 
data can only be obtained after decoder postprocessing 
via path 217. Such postprocessing may include re-scal- 
ing, re-rastering, shifting and repeating pictures from the 
DUT, ther by significantly increasing the difficulty ol as- 
certaining the compliance of the DUT. 
[0027] For example, postprocessing may introduce a 
number of different problems when comparing a decod- 
ed frame from the DUT 1 20 with a decoded frame from 



6 

the r f r nee decoder 1 30. First, due to the unknown 
decoding delay of the DUT, the video output analyzer 
140 must ensure that the comparison is temporally per- 
formed on the correct frames. Second, the proper loca- 

5 tion (vertically and horizontally) of the first pixel must be 
determined due to cropping operations, e.g. : which line 
is the top line and which pixel is the first pixel of the line. 
Third, the proper frame size (vertical size and horizontal 
size) must be determined due to interpolation opera - 

10 tions. Namely, the video output analyzer 140 must de- 
termine the number of display pixels per decoded pixel, 
horizontally and vertically. Fourth, the proper scaling 
must be determined, e.g. : what numerical value corre- 
sponds to black, white, and colors after the postprocess- 

15 ing operations. These are only some examples of the 
problems that are introduced by the various post- 
processing operations. However, embodiments of the 
encoded test bitst reams of the present invention are 
preferably especially designed to address these post- 

20 processing problems, and to provide the ability to deter- 
mine the compliance of a DUT. 

[0028] The proof of compliance involves comparing 
the output signal of the DUT 1 20 with the reference vid- 
eo decoder 130 having an equivalent postprocessing 

25 section 21 2. If the postprocessing methods of the post- 
processing section 210 are known and available, then 
the embodiments of the present invention will simply ap- 
ply the same postprocessing to the output signals from 
the reference video decoder 130. In turn, the postproc- 

30 essed frames from both postprocessing sections 210 
and 212 are received by the video output analyzer 140, 
where on a pixel-by-pixel and frame-by-frame basis, the 
R, G and B matrices are differenced, and the magnitude 
of the error is computed via a R compare section 230, 

35 G compare section 232 and B compare section 234, re- 
spectively. 

[0029] However, such information may not be availa- 
ble and the encoded test bitstreams must be designed 
to assist the video output analyzer 140 in ascertaining 
40 the functions of the postprocessing section 210. Name- 
ly, by analyzing the output signal from the postprocess- 
ing section 210, the video output analyzer 140 should 
be able to calibrate for brightness, contrast, and pixel 
location. 

45 [0030] More specifically, "intra-coded areas" are in- 
serted into one or more frames (e.g., a final reference 
frame at the end of a group of pictures (GOP)), in the 
lest bitstreams which show the scale and brightness 
(and/or color) range. For example : FIG. 3 illustrates a 

50 test frame 300 having at least one intra coded area 310 
and an actual test data area 320 (multiple intra coded 
areas can be applied, e.g., a second intra coded area 
can be applied below the actual data area). The intra 
coded area 31 0 is constructed with known reference lev- 

55 els 330 (e.g., absolute or difference levels) with scale of 
brightness and color of known values. These referenc 
levels are, in turn, encoded/d coded using discr te co- 
sine transform (DCT)/inverse discrete cosine transform 
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(IDCT) to produc brightness and color valu s. Since 
both the intra coded area and the actual test data un- 
dergo the same encoding/decoding and postprocessing 
processes, it is possible for the video output analyzer 
140 to assess the luminance and color scale of the ac- 
tual test data by using the reference levels in the intra 
coded area as a template or look-up table. Once the lu- 
minance and color scale of the actual test data is as- 
sessed, it is then possible to determine whether any ex- 
cursion is greater than what would have been a magni- 
tude of 1 as specified in the MPEG standards. 
[0031] The above intra coded areas may further in- 
corporate a sequence of a black macroblock, a white 
macroblock each with zero (no color) values for Cr and 
Cb, a macroblock with a particular value of Cr and zero 
value of Cb, and a macroblock with a zero value of Cr 
and a particular value of Cb at the end of each row. The 
video output analyzer can use this sequence to measure 
the luma scale (number of steps between white and 
black and the value of black) as well as the chroma 
scales. To illustrate, luminance scale can be calculated 
by taking the difference of received luminance values 
from the black and white macroblocks, and normalizing 
the output scale to the encoded scale difference (e.g., 
240-16). Similarly, the white chrominance scales can be 
determined by normalizing the chroma outputs during 
the Cr (or Cb) macroblock and the white or black mac- 
roblock. 

[0032] The mapping (frame sizing) of the pixels is ad- 
dressed by detecting the number of pixels between two 
predefined reference pixels, e.g., the first white pixel in 
each of the two white macroblocks, and dividing by the 
expected scale. This provides the scale for re-rastering 
the decoded test frame(s). 

[0033] For example, if two reference pixels (or mac- 
roblocks) are placed 300 pixels apart in the test frame, 
and the decoded frame after postprocessing reveals 
that there are 600 pixels between the reference pixels, 
then the scale is 2. Namely 600 is divided by 300 indi- 
cating that there are twice as many display pixels as de- 
coded pixels. Using this calculated scale, the video out- 
put analyzer 140 can transform or re-raster the post- 
processed frame to produce the original decoded frame 
from the DUT 120. 

[0034] Alternatively, a macroblock 340 having four 
subblocks in a checkerboard pattern is illustrated in FIG. 
3. This checkerboard pattern can be used to detect the 
pixel locations more accurately, instead of using two 
white pixels or macroblocks. Furthermore, the checker- 
board pattern allows for detecting both vertical and hor- 
izontal scaling. The checkerboard macroblock is made 
by coding alternate blocks white and black. 
[0035] Proof of compliance is achieved when the ab- 
solute or root mean square (RMS) difference between 
the xpected decoded frame from the r ference video 
decoder 1 30 and th r -rast red and/or r seal d DUT's 
output frame is within th tolerances as specified by the 
MPEG standards and th IEEE standard for IDCT How- 



ever, in this situation, th difference must subtract any 
inaccuracies caused by the re-rastering or r -scaling of 
the decoded frames from the DUT 120. Namely, the vid- 
eo output analyzer 140 must account for the inaccura- 

s cies contributed by the postprocessing section 210. 
[0036] The inaccuracies of the postprocessing sec- 
tion 210 may be different for different DUTs. As such, 
the inaccuracies must be measured at the time when 
the compliance test is being performed. 

10 [0037] One illustrative method of measuring the inac- 
curacies is to apply the calculated scales (derived from 
re-rastering or re-scaling of the decoded frames from 
the DUT) to the decoded frames of the reference video 
decoder 130. This process mimics the postprocessing 

is of 210, but instead, is applied to the decoded frames 
from the reference video decoder 130. Next, the re-ras- 
tering or re-scaling operations are immediately reversed 
to return the decoded frames from the reference video 
decoder 1 30 to their original state. In other words, apply 

20 postprocessing to the decoded frames from the refer- 
ence video decoder 1 30 and then immediately apply th 
reverse postprocessing. Since the exact state of the de- 
coded Irames from the reference video decoder 130 is 
previously known, it is possible to compare the "de-post - 

2S processed" frames with the B pre-postprocessed" 
frames, thereby measuring the distortion caused only 
by the postprocessing operation itself. 
[0038] It should be noted that even if a DUT is deemed 
to be non-compliant based upon the criteria discussed 

30 above, there is a possibility that the non-compliant judg- 
ment is caused by the postprocessing section. As such, 
if the absolute or root mean square (RMS) difference 
between the expected decoded frame from the refer- 
ence video decoder 1 30 and the re-rastered and/or re- 

35 scaled DUT's output frame is within the tolerances as 
specified by the MPEG standards plus the measured in- 
accuracies caused by postprocessing section, then the 
DUT is deemed to be "possibly compliant". 
[0039] Similarly, if the absolute or root mean square 

40 (RMS) difference between the expected decoded frame 
from the reference video decoder 1 30 and the re-ras- 
tered and/or reseated DUT's output frame is not within 
the tolerances as specified by the MPEG standards plus 
the measured inaccuracies caused by postprocessing 

45 section, then the DUT is deemed to be "non-compliant". 
Namely, if the comparison of the two decoded frames is 
based on a new set of tolerances, which is the sum of 
the inaccuracies contributed by the postprocessing sec- 
tion and the restriction set by the MPEG and IEEE 

so specs, and the DUT's output still does not pass, then 
non-compliance is proven. 

[0040] Furthermore, it is important that the decoded 
frames be synchronized properly before differencing. 
Synchronization of the decoded frames is especially im- 
55 portant wh n a DUT is a hardware devic . Unlike a hard- 
ware DUT th throughput delays of a software imple- 
mented DUT is generally known, sine the output de- 
coded fram s ar in the form of files. Any frame reor- 
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dering, e.g., when B frames are present in the bitstream, 
must be accomplished similarly by bolh decoders 120 
and 1 30. 

[0041] Frames synchronization can be achieved by a 
number of methods, such as knowing the throughput de- 
lay of the DUT in advance and then waiting for a period 
of time equal to this delay before the first frame is cap- 
tured. To synchronize pixels within a Irame. the horizon- 
tal and vertical synchronization pulses are monitored at 
the output of a hardware MPEG video decoder IC to de- 
duce the start of the first active pixel of the first active 
line. 

[0042] Another method of frame and pixel synchroni- 
zation involves the use of "key blacks' within test 
frames. These blocks are designed to contain, within a 
predefined region of a test frame, c g . an intra-coded 
area, an image pattern which is easy to detect both in a 
digital bitstream and in a decoded video signal. Key 
blocks should also be highly compressible in the select- 
d encoding scheme. The key block can be placed at 
the start of each test frame oi only at the lust test frame 
to be compared. Additionally, indexed key blocks can be 
used similarly to a product bar code to identify the frame 
under test. The key blocks used with a matched filter 
would also provide pixel synchronization signal. Key 
blocks are especially useful when tho output frame dis- 
play rate does not match the encoded frame rate as in 
the case of MPEG display on a 72 frame-per-second 
computer monitor. 

[0043] FIG. 4 illustrates one example of a video test 
frame 400 containing key blocks 410 The first two 
blocks of the coded frame are set aside for keying. In 
the preferred embodiment, each block is an 8x8 block 
of pixels, e.g., one quarter of a MPEG macroblock, but 
it should be understood that other block sizes can be 
used. The DCT coefficients for these two blocks are all 
zero except for selected horizontal -only coefficients. 
These coefficients can be chosen to grve an easily de- 
tected pattern in a captured frame or a component or 
composite video signal. For example, the key blocks 
410 are illustrated with a pattern thai starts with white 
and then gradually turns into black, i c each macrob- 
lock only contains a single frequency 
[0044] In fact, other forms of ma chine -readable num- 
bers can be inserted in a row of intra-coded macroblocks 
with a binary encoded frame number The number can 
move with pan-scan so as to ensure that it is always in 
the displayed area. Furthermoie. a line other lhan the 
top or bottom line can be used to ensure that the syn- 
chronization pixels are "on screen" to avoid having the 
key blocks being clipped by the display process. Thus, 
the selective use of reference data, c g . reference lev- 
els, patterned blocks or koy blocks serves to assist the 
present invention in addressing the criticahty of inaccu- 
raci s introduced by the postprocessing of a DUT. 
[0045] It has been found that the I DCT process within 
a DUT often contributes a significant amount of errors 
in the decoding process. Furthermore, a compliance- 



only test may not reveal th xtent of th non-compli- 
ance. As such, the present test bitstream is designed 
not only to assess the compliance of a DUT, but it is also 
designed to gain additional insight into the accuracy of 
s the I DCT process of a DUT 

[0046] Typically, if the IDCT of the DUT is compliant, 
then all errors for all pixels in all frames should be less 
than or equal to 1 in magnitude. If an error is larger than 
this limit, it can be attributed to one or both of the follow- 

io ing considerations. 

[0047] First, when the IDCT of a DUT is non-compli- 
ant, i.e., its accuracy is outside the bounds set by the 
MPEG standards and other referenced documents, 
these types of errors are typically limited to isolated, sin- 

is gle-pixel errors in the difference image. As an alternative 
embodiment, the present method measures the RMS 
error for the entire block and/or macroblock. This meth- 
od can provide information about the DUT's IDCT proc- 
ess and its inaccuracies. 

20 [0048] Second, some function in the DUT is imple- 
mented incorrectly, e.g., the DUT is performing fram 
IDCT when it should be performing field IDCT. These 
are relatively large errors in the decoded video and may 
indicate functional implementation flaws. Typically, er- 

25 rors that are much larger than 1 in magnitude, and 
spread over large areas in the image (e.g., 16x16 pixels 
regions or larger), are indicative of functional implemen- 
tation flaws. 

[0049] Furthermore, the MPEG predictive coding 

30 methodology fosters dependency in the frames within 
the bitstream in exchange for compression efficiency. 
However, an undesirable side effect of such dependen- 
cy is that IDCT errors can often accumulate over time. 
These IDCT errors affect the compliance testing proc- 

35 ess such that it can obscure the flaws of a non-compliant 
DUT Such accumulation can be prevented and ad- 
dressed using different methods. 
[0050] First, the reference decoder 1 30 can be mod- 
ified to use the anchor frames, e.g., I or P frames, from 

40 the DUT. By using the same anchor frames, the IDCT 
errors, if present, will typically accumulate in similar de- 
grees in both decoders. Namely, by starting with the 
same IDCT errors and using the differencing operation 
as discussed above, the cumulative effect of the IDCT 

45 errors is removed. In this fashion, cumulative IDCT er- 
rors can be eliminated as a possible cause for non-com- 
pliance of a DUT. 

[0051] Second, the reference video decoder 1 30 can 
be modified to use the same IDCT algorithm as the DUT. 

50 This method is especially appropriate if the DUT is a 
software decoder where the IDCT process is typically 
defined as a function that can be isolated and copied. 
This method is oven more effective in eliminating cumu- 
lative IDCT errors than the above method, because th 

55 entire IDCT proc ss is duplicated. IDCT errors will ac- 
cumulate in both decoders in exactly the same manner, 
and such errors can be effectively removed by th dif- 
ferencing operation. In turn, the IDCT algorithm can be 
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separately checked for compliance. For bitstreams that 
use the "predictiv test" methodology, e.g., bitstreams 
that are designed such that decoding errors are propa- 
gated from frame to frame, only the last frame would 
have to be compared. A detailed discussion concerning 
a method of using only a subset of frames to perform 
compliance testing is provided below. 
[0052] Third, a limit can be placed on the number of 
frames in each test bitstream (e.g., no more than two 
frames per test bitstream), thereby limiting the severity 
of any cumulative errors. This method is simple to im- 
plement without having to modify the reference video 
decoder 1 30. 

[0053] FIG. 5 and FIG. 6 illustrate two embodiments 
of displays for displaying the errors detected by the vid- 
eo output analyzer 140. These displays provide a sum- 
mary report on the detected errors in an easily viewable 
format via display 1 50. On a pixel-by-pixel basis, the dif- 
ference errors are computed. For example, a single 8x8 
block of pixel differences is shown in a numerical format 
(FIG. 5) and in a color-coded format (FIG. 6). 
[0054] FIG. 5 illustrates a 8x8 block 500 of pixel dif- 
ferences 510, 520 and 530. The zero (0) difference val- 
ues 510 indicate that there are no difference between 
the DUT's output signal as compared to the reference 
video decoder's output signal at these pixel locations. 
The difference values 520 indicate that there are some 
differences between the DUT's output signal as com- 
pared tothe reference video decoder's output signal, but 
the differences are within acceptable limits, e.g., not ex- 
ceeding the absolute magnitude of 1 at these pixel lo- 
cations. Finally, the difference values 530 indicate that 
there are significant differences between the DUT's out- 
put signal as compared to the reference video decoder's 
output signal, where the differences exceed acceptable 
limits. This type of representation is the finest represen- 
tation of errors, since it is pixel based. It can be displayed 
as numbers by the video output analyzer 140 (or on a 
logic analyzer), or as an image on a video monitor 1 50. 
[0055] To visually enhance the errors, FIG. 6 illus- 
trates a Bx8 block 600 of pixel differences in a color dis- 
play format. For example, zero errors can be displayed 
as gray 610, errors with magnitude of 1 or less can be 
displayed as yellow 620 (a warning, but still compliant), 
and errors greater than 1 can be displayed as red 630 
(non-compliant). These images could be one form of the 
Y, Cr : Cb "results" as shown in FIG. 2. 
[0056] Alternatively, the displaying methods can be 
expanded to the block and macroblock level. Namely, a 
numerical value or color (or shades of gray) can be se- 
lected to represent "no error", a "slight acceptable error" 
and "non -compliant error" for a block or macroblock. 
[0057] Another alternative method of representing the 
"result" is to simply accumulate the number of errors in 
each f ram , or ov r the ntir sequenc , that xc ed 1 
in magnitud . If any rror exce ds 1 for any pixel in the 
decod d video s quence, the DUT is consid red non- 
compliant. This type of result representation is less in- 



formative than the pix l-bas d result as d scribed 
above, but it is simpler to implement. 
[0058] However, the above pixel-by-pixel and frame- 
by-frame approach is computationally expensive and 

5 has a large storage requirement. To illustrate, if each 
test bitstreams has over 1 00 frames, and each decoded 
frame (at 720x480 images size, 4:2:0 chroma format) 
requires about 0.5 MB of storage space, then it is nec- 
essary to store over 50 MB of reference frames to per- 

10 form each test. 

[0059] A third embodiment of the present invention 
employs a method for analyzing the video output of 
MPEG-like video decoders using only a subset of 
frames in the bitstream. The advantage of this method 

is is that only a subset of frames are stored and compared, 
thereby reducing storage requirements and computa- 
tional overhead. 

[0060] More specifically, this subset frame analysis 
takes advantage of the predictive nature of the encoded 

20 test bitstreams. MPEG compliant bitstreams typically 
employ motion estimation and motion compensation to 
remove redundancy that exists between the consecu- 
tive frames (or pictures) of a video image sequence. In 
brief, a current frame can be reconstructed from a ref- 

25 ©rence frame and the difference between the current 
and reference frames by using the motion information 
(motion vectors). The reference frames can be a previ- 
ous frame (P-frame), or previous and/or future frames 
(B-frames). 

30 [0061] In one embodiment, the input test bitstream 
contains only a sequence of P frames following an I 
frame without any B frames. For such tests, only the last 
frame of the test or the last frame of a sequence of 
frames (a P frame) from both decoders 120 and 130 is 
35 evaluated and compared for compliance. For example, 
the DUT 120 is compliant if the peak error magnitude is 
not greater than 1 for the last P frame as compared to 
the last P frame of the reference decoder 1 30. This sub- 
set frame analysis is premised on the fact that errors 
40 that appear in earlier P frames are propagated down to 
the last P frame. As such, the DUT is compliant only if 
the decoder performed all the functions under test cor- 
rectly, and where the IDCT mismatch is insignificant. 
[0062] Furthermore, the subset frame analysis may 
45 provide additional insight into the DUT. To illustrate, if 
the last frame of the DUT is captured and compared, 
isolated single-pixel errors are most likely due to effect 
of accumulated IDCT mismatch (error), whereas larg - 
area errors represent implementation flaws and there- 
to fore failure of the particular test. 

[0063] However, for tests that may require B frames, 
the method may store a sequence of one or more r - 
constructed B frames, capture the corresponding 
frames from the DUT, and perform the error analysis on 
55 the captured B frame. B frames require sp cial tr at- 
ment because, unlik P frames, the B fram s do not 
propagate rrors forward in time. Namely, if an error oc- 
curred within a B frame, then only an analysis of that 
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particular B fram will reveal the error. 
[0064] For example, lor a sequenc of "I.RRRRRP" 
frames, the subset analysis method may simply com- 
pare the last P frame to detect compliance of the DUX 
For a sequence of "l,RRRB t RRP" frames, the subset 
analysis method should compare the last P frame plus 
the B frame to ensure proper evaluation of the DUT for 
compliance. 

[0065] FIG. 7 is a flowchart that illustrates an embod- 
iment of a compliance testing method 700 of the present 
invention. The method starts in step 705 and proceeds 
to step 710. where the test bitstreams are decoded by 
both the DUT 120 and the reference decoder 130. As 
discussed above, the decoding process for the refer- 
ence decoder 1 30 can be optionally implemented in ad- 
vance, where the decoded frames are stored for retriev- 
al at a later time. 

[0066] In step 720, method 700 can optionally select 
a subset of the decoded frames for evaluation. The se- 
lected frames for comparison depend upon the type of 
the test frames, e.g., whether there are P and/or B 
frames within the test sequence as discussed above. If 
a subset of the frames is not selected, then the compli- 
ance testing is conducted for all decoded frames. 
[0067] In step 730, method 700 queries whether the 
decoded frames have undergone postprocessing. If the 
query is negatively answered, then method 700 pro- 
ceeds to step 735, where on a pixel-by-pixel and frame- 
by -frame basis, the Y, Cr and Cb matrices are differ- 
enced, and the magnitude of the error is computed. If 
the query at step 730 is affirmatively answered, then 
method 700 proceeds to step 740, where the post- 
processing inaccuracies are computed. 
[0068] In step 750, method 700 queries whether the 
difference result is within the MPEG limits minus the 
computed postprocessing inaccuracies. If the query is 
affirmatively answered, then method 700 proceeds to 
step 755, where the DUT is deemed to be compliant and 
the method ends in step 775. If the query is negatively 
answered, then method 700 proceeds to step 760. 
[0069] In step 760, method 700 queries whether the 
difference result is within the MPEG limits plus the com- 
puted postprocessing inaccuracies. If the query is af- 
firmatively answered, then method 700 proceeds to step 
765, where the DUT is deemed to be possibly compliant 
and the method ends in step 775. It the query is nega- 
tively answered, then method 700 proceeds to step 770, 
where the DUT is deemed to be non-compliant. The 
method ends in step 775. 

[0070] Although various embodiments which incorpo- 
rate the teachings of the present invention have been 
shown and described in detail herein, those skilled in 
the art can readily devise many other varied embodi- 
ments that still incorporate these teachings. 



Claims 

1 . A method of evaluating a decoder under test (DUT), 
comprising the steps of: 

5 

(a) sending a test bitstream to the DUT, where 
said DUT generates a DUT output signal hav- 
ing one or more decoded test frames; 

(b) providing a reference output signal having 
10 one or more decoded test frames; and 

(c) comparing, frame by frame, said DUT output 
signal and said reference output signal to eval- 
uate the performance of the DUT. 

is 2. The method of claim 1 1 wherein said providing step 

(b) comprises the step of sending said test bit- 
stream to a relerence decoder, where said refer- 
ence decoder generates said reference output sig- 
nals. 

20 

3. The method of claim 1 or 2, further comprising the 
step of: 

(a') computing a postprocessing inaccuracy 
of a postprocessing process associated with said 
25 DUT before comparing said DUT output signal and 

said reference output signal. 

4. The method of claim 1, 2 or 3, further comprising 
the step of: 

30 (b 1 ) selecting only a subset of said decoded 

test frames from said output signals for said com- 
paring step (c). 

5. The method of claim 4, wherein said test bitstream 
35 comprises at least a sequence of P frames, where 

said selected subset of frames comprises at least a 
last P frame from said sequence of P frames. 

6. The method of claim 1, 2, 3, 4 or 5, wherein each 
40 of said output signals comprises a luminance and 

chrominance signals, where said comparing step 

(c) comprises the step of differencing said lumi- 
nance and said chrominance signals between said 
output signals on a pixel-by-pixel basis, to produce 

45 a difference error value for each of said pixels. 

7. The method of claim 6, wherein said DUT is deemed 
to be compliant, if said difference error value minus 
said computed postprocessing inaccuracy is within 

so a predefined acceptable limit. 

8. The method of claim 6 or 7, wherein said DUT is 
deemed to be possibly compliant, if said difference 
error value is within a pr d fin d acceptabl limit 

55 ptus said computed postprocessing inaccuracy. 

9. The method of any preceding claim, further com- 
prising the step of: 
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(a") incorporating at I ast one intra-coded ar- 
ea within at least one of said test frames. 



1 0. The method of claim 9, where said intra-coded area 
contains one or more reference levels. s 

11. The method of claim 9 or 10, where said intra-coded 
area contains one or more key blocks. 

12. An apparatus for evaluating a decoder under test 10 
(DUT) by applying a test bitstream, said apparatus 
comprising: 

a reference source, for providing a reference 
output signal having one or more decoded ref- is 
erence test frames, where the test bitstream is 
received by the DUT to generate a DUT output 
signal having one or more decoded DUT test 
frames; and 

an analyzer coupled to said DUT and said rel- 20 
erence source, for comparing, frame by frame, 
said DUT output signal and said reference out- 
put signal to evaluate the performance of the 
DUT 

25 

13. The apparatus of claim 12, wherein said reference 
source comprises a reference decoder, for receiv- 
ing the test bitstream to generate said reference 
output signal. 

30 

14. The apparatus of claim 12 or 13, wherein said an- 
alyzer computes a postprocessing inaccuracy of a 
postprocessing process associated with said DUT 
before comparing said output signals. 

35 

15. The apparatus of claim 12, 13 or 14, wherein said 
analyzer selects only a subset of said decoded test 
frames from said output signals for comparison. 

16. The apparatus of claim 1 2, 1 3, 1 4 or 1 5 wherein the *o 
test bitstream contains a plurality of test frames, 
where at least one intra-coded area is incorporated 
within at least one of said test frames. 

17. A bitstream having one or more encoded test *s 
frames stored on a recordable medium, for testing 

a decoder under test (DUT), said bitstream com- 
prising: 

at least one test frame having an intra-coded 
area containing one of more reference data and a so 
test area containing test data, where said bitstream 
is designed to be forwarded to both the DUT and a 
reference decoder for evaluating the DUT 

18. The bitstr am of claim 17, wher in said r f rence ss 
data compris s one or mor r fer nc I vels, or 
one or more key blocks. 
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Testing a video decoder 







(57) An encoded bitstream for a generator (110) is 
applied to a black-box video decoder (120) to evaluate 
the performance of the decoder (1 20). The encoded bit- 
stream is designed such that the output from the decod- 
er (120) can be evaluated without the need of a human 
observer. The encoded bitstream is applied to both a 



decoder under test (DUT) (120) and a compliant refer- 
ence decoder (130) and a video output analyzer (140) 
compares the results from both decoders (120, 130) to 
determine compliance of the DUT (120). Additional em- 
bodiments address postprocessing and the selection of 
a subset of test frames for evaluation. 
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